Evaluating autonomous ground-robots

The robotics community benefits from common test methods and metrics of performance to focus their research. As a result, many performance tests, competitions, demonstrations and analyses have been devised to measure the autonomy, intelligence, and overall effectiveness of robots. These range from robot soccer (football) to measuring the performance of a robot in computer simulations. However, many resultant designs are narrowly focused or optimised against the specific tasks under consideration. In the Multi-Autonomous Ground-robotic International Challenge (MAGIC) 2010 the need to transition the technology beyond the laboratory and into contexts for which it had not specifically been designed or tested meant that a performance evaluation scheme was needed that avoided domain-specific tests. However, the scheme still had to retain the capacity to deliver an impartial, consistent, objective and evidence-based assessment that rewarded individual and multi-vehicle autonomy. It was also important to maximise the understanding and outcomes for technologists, sponsors and potential users gained through after-action review. The need for real-time, simultaneous and continuous tracking of multiple interacting entities in an urban environment and over 250,000 square metres in real time compounded the complexity of the task. This paper describes the scheme used to progressively down-select and finally rank the teams competing in this complex and ‘operationally realistic’ challenge.